Mainstream adoption of mixed reality technologies including virtual reality (VR) and augmented reality (AR) are poised to alter many industries including healthcare. Applications such as the radiological viewing of medical scans as well as surgical planning may be revolutionized by these technologies. One impediment to wide-scale utility and adoption is that current methods of three dimensional (3D) rendering focus on visualizing surfaces, outer voxels, or maximum intensity voxels while the majority of information in the interior of a 3D volume is not visualized.
For 3D volume values generated by techniques such as magnetic resonance imaging (MRI), computed tomography (CT), microCT, or other thin slice tomography (thin 2D slices of a 3D volume) scanning methods, in current practice, radiologists view the entire series of 2D slices and then must mentally reconstruct 3D spatial relationships. Due to the difficulty of this mental task, effort has been made to better visualize 3D volumes and spatial relationships. These include 1) direct volume rendering (DVR) in which a 2D projection of a 3D dataset is rendered, 2) isosurface rendering using a marching cubes algorithm to generate a surface model and 3) 3D model projections with maximum intensity projection. In DVR, every voxel is mapped to a specific opacity and color with a transfer function and using either volume ray casting, splatting, shear warp, or texture based volume rendering, and a 2D projection of a 3D volume is generated. These methods are often computationally expensive and generate an accurate visualization of surfaces of interest in the 3D volume but result in a 2D image rather than a 3D rendering and so cannot be rotated and viewed from different angles, and will be a flat 2D image even in a mixed reality environment. Similarly, maximum intensity projection shows voxels with maximum intensity that fall in the way of parallel ray traces drawn from the viewpoint and generates a 2D image from a 3D object. One 3D object that works in mixed reality is a mesh of the surface of a volume. An isosurface mesh renderings of a 3D volume can be generated with a marching cubes algorithm as a series of polygonal surface meshes or with volumetric meshes, with the caveat being that at any given time, only the small percentage of the voxels at the surface of a volumetric mesh is rendered or easily visible.
Usually, when multiple surfaces are overlaid atop of one another, to be visible, transparency is added to surface models to visualize surfaces of objects within objects, or point clouds are used, notwithstanding occlusion issues that do not allow good visualization of interiors. In other words, even with transparency, only a small percentage of the voxels at the surface of a 3D volume out of the total voxels in a 3D volume are visible either because they are the only ones rendered such as in surface models, or because they are the only ones not occluded with point clouds. Further, surface renderings require identification of hard edges (surface of lesion needs to be identified for example), which is often hard to determine in clinical scans in which the boundaries between diseased and healthy tissues are not well defined nor well demarcated. This is especially true in poor resolution scans like CT where a voxel may contain both healthy and diseased tissue and so is hard to classify as either healthy or diseased. In other words, many times lesions are visually resolvable as a gradation in scalar values and have subjectivity involved. This is a challenge for automated segmentation and computer vision as well, as algorithms often perform poorly compared to an experienced human radiologist who can compensate for the lack of hard edges by using a subjective clinical judgment of the visualized data. Even in good resolution scans, often structures of interest are interwoven with one another with unclear boundaries making it hard to represent the data with a 3D surface.
For this reason, 2D slices are often combined to view cross-sections of the 3D surface volume. Hence, different visualization techniques may be combined to present more complete visual understanding of a scanned volume. Again however, at any given time, only a small percentage of total voxels in a volume are visualized either because they are in that 2D slice (a small percentage of interior voxels) or at the surface for the 3D surface object (only exterior voxels).